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SIMIARY 



Two tests of psychoBotor ability (hand-eye coordination) previously developed by the Air 
Force Huaan Resources Laboratory were validated against USAF Undergraduate Pilot Training (UPT) 
perfomance. The two tests significantly predicted UPT eliminations and differentiated between 
the better and weaker graduating students. Various operational Iroplenentatlon strategies for the 
psychonotor tests were evaluated. Including using the tests by themselves as an additional 
selection gate and combining the scores from the tests with other nonwlly available selection 
Infoniatlon. The combined approach proved most accurate, and three Integrated Pilot Candidate 
Selection Models (IPCSMs) were developed using combinations of the psychomotor tests. Air Force 
Officer Qualifying Test scores, grades from the Flight Sct^nlng Program and biographical data. 
Con)«rtsons wong the IPCSMs were made and recommendations for Implementation were presented. 
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UTILIZATION OF PSYCHOMOTOR SCREENING FDR USAF PILOT CANDIDATtS: 
INDEPENDENT AND INTEGRATED SELECTION METHODOLOGIES 



I. INTRODUCTION 

A multi-year research and development (R«0) program was designed at the request of the United 
States Air Force (USAF) Air Training Conmand (ATC) to Improve procedures for the selection of 
candidates fo*" USAF Undergraduate Pilot Training (UPT). The overall Rtt program Is described In 
Kantor and Bordelon (1985). Two Important objectives of this research were to capitalize on 
state-of-the-art technologies In computer-aided testing for the development of accurate measures 
of psychomotor ability (hand-eye coordination) and then to Investigate different w^ys to use 
these psychomotor measures In the selection of UPT candidates. 

The principal goal of selection for UPT Is to screen out those candidates with extremely low 
chances of completing training. According to ATC calculations, the average cost of each UPT 
ellinlnee Is approximately J64.000 (FY84 dollars). Therefore, screening out potential failures 
can avoid considerable attrition costs. Also, the Increased complexity of the weapon systems and 
missions In which USAF aircrews are employed requires selection of high-quality students. Flying 
aptitude tests are tradUlonally used to differentiate among UPT candidates. 

The Ideal selection system would measure and consider all relevant characteristics of a 
candidate. These would Include physical and mental characteristics as well as the candidate's 
previous experiences, such as actual flying. This Information would be used to predict a 
candidate's probability of success In an optimally weighted manner. The candidates with the 
lowest probability of success would be screened out before training. 

The UPT selection system currently used by the USAF does not consider all relevant 
Information about candidates. Thus, mar\y candidates fall training because of deficiencies In 
flying aptitude. Psychomotor ability Is one of several characteristics that historically has 
demonstrated relevance to flying performance. Though not currently used by the USAF, psychomotor 
screening Is used by several other national Air Forces to select their pilot candidates. This 
report documents the validation of two tests of psychomotor ability against USAF PT performance 
and develops strategies to Incorporate this Information Into the USAF pilot selection system. 



Background 

Between February 1942 and July 1955, measurement of the psychomotor ability of pilot training 
candidates was an Important component of the Air Force (prior to 1947, the Amy Air Corps) pilot 
election procedure. Screening on psychomotor ability was discontinued In 1955 because the 
alectro-mechanlcal equipment used to measure It was difficult to calibrate under decentralized 
testing conditions and was hard to maintain (Passey A McLaurIn, 1966). Since 1955, the principal 
components considered In USAF pilot selection have Included physiological fitness, the 
paper-and-pencll Air Force Officer Qualifying Test (AFOQT), and actual flying experience. 
Recently, the candidate's age and possession of a technical degree (defined as a college degree 
In math, science, engineering, or computer science) have been Included as factors In the 
selection decision. 

Improvements In computer technology provided the potential to develop reliable and 
maintainable psychomotor tests and revived Interest In psychomotor assessment for pilot candidate 
screening. In 1969, the Air Force Human Resources Laboratory (AFHRL) contracted to design, 
develop, and determine the reliability of two computer-based psychomotor tests (Sanders, 
Valentine, « McGrevy, 1971). The basic task elements of these new tests and the responses 
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required of the subject duplicated earlier electro-mechanical tests developed In 1942 by the Anny 
Air Forces School of Aviation Medicine (now the USAF School of Aerospace Medicine). The newly 
designed tests differed from the earlier versions In several respects, but the most Important 
difference was that, for the first tiBie, a mlnlcomptcer controlled the test procedures, providing 
reliability and standardization not achievable on electro-mechanical devices. 

The first of these tests, called Two-Hand Coordination, Is a pursuit tracking task measuring 
the subject's ability to coordinate the movement of both hands to track a moving target. The 
second test, called Complex Coordination, requires multi-limb, dual-task compensatory tracking 
and measures the subject's ability to adjust two Independent responses to control a changing 
stimulus pattern. The effectiveness of the tests was Initially demonstrated with a sample of 120 
Officer Training School (OTS) cadets. Stable task performance was obtained after 10-m1nute test 
sessions and rel1ab11^'\»s were very high. Inter-test correlation was low, suggesting that each 
test may contain unique Information and that both should be further evaluated. The high multiple 
correlation between the scores on both tests and training outcome at UPT (R ■ .46) suggested that 
these psychomotor tests could enhance the prediction of pilot training success. These results 
precipitated a recommendation for large-scale validation of an operational version of the 
psychomotor tests (McGrevy A Valentine, 1974). 

In 1975, AFHRL contracted with Systems Research Laboratory, Inc., Dayton, Ohio, to design and 
develop five computer-based test devices that could administer the two psychomotor tests. These 
transportable devices were self-contained and did not require highly trained test 
administrators. The devices faithfully reproduced the two tests that BloTechnology, Inc. had 
developed on the minicomputer. 



Objecti ves 

In 1978, AFHRL began a multi-year RAO effort to use the new test devices to obtain 
psychomotor measures from USAF pilot candidates from all three conmlsslonlng sources: Air Force 
Acadefny (AFA), Air Force Reserve Officer Training Corps (AFROTC), and Officer Training School 
(OTS). Initially, the actual UPT outcomes of the candidates would be used to validate the 
usefulness of the psychomotor scores for pilot screening. (Though not part of this report, these 
test scores will later be validated against measures of successful operational flying.) In 
addition, this stu^y sought to determine the best manner to Integrate the psychomotor test 
Information with available selection Information to achieve the greatest Improvement In the USAF 
pilot selection process. 



lU METHOD 



Predictive Validation 

A testing plan was jointly developed between AFHRL, ATC, AFA, and AFROTC. in the spring of 
1978, four of the psychomotor test devices were transported to the Air Force Academy to test the 
cadets from the three upper classes. In the spring of 1979, the devices were shuttled by van to 
all of the AFROTC detachments In the contlnenUl United SUtes to test AFROTC juniors and 
seniors. Then, the devices were placed at the AFHRL Lackland Testing Facility, Lackland AFB, 
Texas, to test OTS pilot candidates entering the Flight Screening Program (FSP) from the fall of 
1979 throrgh the fall of 1981. The pilot candidates were then tracked through UPT, and the 
performance of those who completed UPT between FY79 atid FY82 was used to validate the psychomotor 
test scores. All UPT student records from FY79 to FY83 were obtained to ensure as many matching 
cases as possible. The test subjects who completed UPT during FY83 were used to cross-validate 
the developmental results. 
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CoacMrrtwt Villdatlow 



As an addltlOMl validation of tht psjfchoaotor scoras, the ttst devices were sent to Williams 
AFB. Arizona, In Jgnt 1984 to test UPT graduating students. For this sanple, the results of the 
Advanced Training RecoMtndatlon Board (ATRB) wtrt obtained as criteria. The ATR6 reconwiends 
students for fo11o«*on asslgnaents to either fighter-attack-reconnaissance (FAR) or 
tanktr-transport-boabtr (TTB) aircraft, and only ths better students receive a FAR recommendation. 

Table 1 shovs tht saaple sizes of tht test groups. Analyses reported here Include factor 
analysis of tht psychOMtor ntasurts and coa^Mrlsons anong the aeanf for the different training 
outcoM groups from UPT. A piychoaotor screening system and three tegrated selection systems 
wtrt dtvtloptd using multlpit linear rtgrtsslon ttchnlques. The utility of the systems was 
demonstrated using a hit-miss classification algorithm which computed correct and Incorrect 
selection dtclslons for all of tht posslbit cutoff scorts. 



Tabit 1. Suhjtcts Ttsttd by Sourct 



Soyrct Subjtcts tested 



Air Force Academr 


382 


Air Force Reserve Officer Training Corps 


1,229 


Officer T.alning School 


603 


Unknown Source of Commission 


314 


Testing Plan Total 


2,528 


UPT, Williams AFB 


95 


Total 


2,623 



Apparatus 

The test devices admlnlsttrtd both of the psychomotor tests. The Internal computer generated 
the test Images and presented them on a monochrome cathode-ray tube (CRT). A pre-recorded 
cassette tape playtd ttst Instructions to tht subject through headphones. Two small Joysticks 
(one on the left and the other on the right of the CRT) were used for Inputs during the Two*Hand 
Coordination Test. An alrcraft-s^le, floorHKMinted Jciystick and two foot-operated, rudder-style 
pedals wtrt ustd for Inputs during tht Complex Coordination Test. The subject sat on an ordinary 
Chair In front of the f ret*stand1 ng ttst dtvlct. Located on the rear of the device were three 
digital displays that presented the final scores of either test that was selected by a switch. 
Once sUrted by a Ust administrator, the device would provide the subject with 5 minutes of 
Instructions and practice time for tht first ttst, then a 5-m1nute test session, and then repeat 
the same stqutnct for the second ttst. Thus the entire procedure required only 20 minutes. 



Two^Htnd Coordination Procedurts 

The first of the tests ^ Two-Hand Coordination - presented a triangular-shaped target and a 
cross-shaped pipper on the CRT (Figure 1). The computer moved the target In an elliptical path 
and with varying speeds (faster near the 4 o'clock position and slower near the 11 o'clock 
position). The subject moved the pipper using the two small Joysticks. The left Joystick 
controlled the pipper only In tht up-down, or vtrtlcal, axis whereas the right Joystick 
controlled the pipptr In tht Itft-rlght, or horizontal, axis. The subjects were Instructed to 
use both Joysticks sImulUneously In a coordinated manner to move the pipper, keeping It as close 
as possible to the target on the CRT. 
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PSYCHOMOTOR TESTS 



TEST I 

• LEFT - RIGHT HAND COORDINATION 

• 10 MINUTES 




LEFT 
QrtW 



mOHT 



"gm^ 1- Tt^-ttand CoordlMtfon Test Dtplctlon. 



Sixty tiMS each second, the co^auter Measured In inches the absolute distance froi the 
pipper to the target as both Mved around the CRT. The horizontal and vertical co^)onents of 
this error MasureMnt were c(Mputed and accuaulated during the S-alnute test period. The two 
scores obtained froa Two-Hand Coordination were the emulative horizontal (X,) and cuMjlatlve 
vertical (Y^) error scores. ' 

Comlex Coordination Proceduir»s 

The second test - Co^)lex Coordination - presented a set of cross-hairs centered on the CRT, 
a dot-shaped pipper. and a thin vertical bar at the bottoa of the CRT (Figure 2). Hie subject 
controlled the pipper. both horlzonUlly and vertically, using the fioornwuntcd Joystick. The 
control responses were the reverse of what Is traditionally required on aircraft (I.e.. left 
■oveMnt of the Joystick aoved the pipper to the right, back aovMnt of the Joystick acved the 
pipper downward, etc.). This arrangcMnt was Intentionally selected to reduce the advanUge of a 
subject with prior flying experience. In the saw w*y. the vertical bar was nved horlzonUlly 
to the left by pressing the right foot pedal and to the right by pressing the left foot pedal. 
The subjects were Instructed to use the Joystick to center the pipper horizontally and vertically 
on the intellection of the cross-hairs and siwjltanecusly press the appropriate rudder-style 
pedal to center the rudder bar" over the lower part of the vertical cross-hair. Ihe coq>uter 
randoaly changed the gain on the Input controls, which drove the pipper and rudder bar off their 
■arks, requiring the subject to constantly co^iensate for the forced displaccMnt. 

* l**""" eotP"^"^ eeasurod in inches and accuwlated during the S-ainute 

test period the absolute disUnce frw the pipper to the Intersection of the cross-hairs and fro. 
the rudder bar to the vertical cross-hair. The three scoros obUined f.vw the CoBlex 
Coordination test were the ciaulative horizontal error co^>onent for the pipper (X,). tN 
emulative vertical error co^ionent for the pipper (Vg). .nd the ciMJlative horizonUl error 
for the rudder bar (Zg). 
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PSYCHOMOTOR TESTS 



7E8TI1 

• STICK ft RUDDER SKILLS 

• 10 MINUTES 




Figu re Z. Conplex Coordination Test Depiction* 
III. RESULTS 

P^rchoBOtor Scores 

The two psychOMOtor tests produced a total of five error scores (X], Y], Y2, 
Z2) for each subject* Because the scores neasure error, low scores are better than high 
scores* Tabit 2 Tsts basic statistics for the scores obtained from the original sanple of 2,528 
candidates froa ail coMlsslonlng sources* The Mean error scores fron the Two-Hand Coordination 
Test were Mich higher (worse) than those for the Complex Coordination Test, which suggests that 
the requirements for this pursuit tracking task were nore difficult than for the compensatory 
tracking task* 



Table 2* Neans, Standard Deviations and Ranges of 
PsychOMtor Scores (N > 2,528) 



Test score 


Mean 


SD 


Range 


Two-Hand Coordination 








XI (Horl;) 


14.709 


5,400 


428 - 65,268 


Yl (Vert) 


16,748 


5,538 


5,687 - 57,292 


Complex Coordination 








X2 (Horli) 


4,514 


6,520 


326 - 71 .040 


Y2 (Vert) 


3,860 


6,445 


486 - 76.320 


Z2 (Rudder) 


5,580 


6,018 


167 - 71 ,040 
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Outliers 



ln« values at the high end of the ranges of all five scores were such larger than 
anticipated. Subjects averaged approxlaately 4 inches of trKkIng error for the entlrv Snilnute 
test session (4 Inches x 5 nlnutes x 60 sec per aln x 60Hz - 72.000 Inches). These large values 
night be expected as the error score when the subject Mde no tracking Input on that coivonent of 
the test. HoMever. none of the subjects had extreaely largo values on all five scorvs. If one 
or nore axes were Ignored, the test tcken by that subject would be different froB the presented 
test and the scores would not be coaparable to the results of other subjects. These extreaely 
large scores were frc« 7.3 to 11.2 sUndard deviations above the aean of the i«spect1vo 
dIstHbutlons. There were few such outliers, with the vast Mjorlty of the scorvs within six 
standard deviations of the respective wans. A cutoff of six sUndard deviations above the 
respective iwan was arbitrarily liposed, reaoving 35 cases froa further analyses. The updated 
descriptive statistics Including the cutoff of the new staple of 2,493 subjects are listed In 
Table 3. 



Table 3 . Keens, Standard Dtvlatlons, Ronges, and Cutoffs 
«*f P^ychoaotor Scores (N ■ 2,493) 



Test Score 


Mean 


SO 




Range 


Cutoff 


Two-Hand Coordination 












XI (Horiz) 


14,624 


5,194 


428 


- 39,358 


47,109 


Y1 (¥ert) 


16,683 


5,427 


5,687 


- 45,093 


49,976 


Conplex Coordination 












X2 (Horiz) 


4,113 


4,349 


326 


- 42,215 


43,634 


Y2 (Vert) 


3,426 


3,600 


486 


- 37,667 


42,530 


Z2 (Rudder) 


S.502 


5.746 


167 


- 40.888 


41.688 



Intercorrelatlons 

The five test scores were all Measures of p^choMtor ability; therefore, each could Include 
some coiwon Information about the subjects. The strength of this shared Infonwtlon, expressed 
as intercorrelatlons. Is shown In the Mtrlx at Table 4. As sight be expected, the pvchoMOtor 
scores within the sane test had the highest fnttrcomlatlons whervas the correlations across 
tests were small. 



Table 4. Intercorrelatlons of P^ychOMtor 
Scores (N • 2,493) 





Xl 




X2 


^2 


XI (HoHz) 










Y1 (tert) 


.87 








X2 (Horiz) 


.20 


.25 






Y2 (Viert) 


.19 


.23 


.86 




Z2 (Rudder) 


.20 


.24 


.77 


.71 



A principal coaponents factor analysis followed by a vaHaax roUtlon further evaluated the 
relationships aM>ng the five scores. The two extracted fKtors had eigenvalues greater than 1.0 
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(2.85 and 1,58) and accounted for 88.5% of the original variance. Table 5 lists the rotated 
factor loadings. The varlmax rotation separated the scores clearly Into two factors. Each 
factor was donlnated by only the scores from a single test. Therefore, the two psychomotor tests 
were measuring different abilities with very little In conmon between the tests. The results of 
the factor analyses Indicate that both of the tests should be considered In further validity 
analyses. 



Table 5. Varlmax Rotated Factor Loadings of 
Psychomotor Scort* (N - 2,493) 





h 


Yl 


X2 


^2 




Factor I 


.10 


.15 


.96 


.88 


.79 


Factor II 


.92 


.92 


.11 


.10 


.13 



Criterion Data 

The criterion used to assess the valldltv of the psychomotor test scores was training outcome 
in UPT. Except for the Williams AFB sample, it was necessary to track the tested candidates 
through the 49 weeks of UPT before their outcome was known. Many candidates In the sample failed 
to start UPT because they did not receive a commission or were eliminated during normal 
screening. To obtain criterion data on as large a sample as possible, all of the UPT results 
from FY79 to FY83 were obtained from ATC. Table 6 lists the sizes of the validation samples 
after matching the UPT criterion and psychomotor data. A sample of 1,725 candidates who had 
valid psychomotor test Information and ended UPT between FY79 and FY82 was Identified and 
analyzed In August 1983. This group was used for the predictive validation analyses and the 
development of the psychomotor and Integrated selection systems. A separately matched and more 
recently available sample of 166 FY83 UPT students was analyzed In November 1983 to cross- 
validate these selection systems. Finally, the results of the 95 Williams AF8 students tested 
nearing UPT graduation were analyzed In August 1984 for cc^urrent validity assessment. 



Table .6. Psychomotor Validation Samples 



Sample 


N 


Predictive Validation 


1,725 


(FY79 to FY82 UPT, 5 bases) 




Cross-validation 


166 


(FY83, 5 UPT bases) 




Concurrent Validation 


95 


(FY84 UPT, Williams AF8) 





Predictive Validation 



For the psychomotor scores to be useful, at least some of them must significantly 
differentiate between candidates who graduate from UPT and those who eliminate. Reasons for 
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elimination from UPT are classified as flying deficiency, academic deficiency, medical, 
self -Initiated, and fatality. If psychorriotor ability Is related to flying aptitude, then the 
greatest differences should exist between the psychomotor scores of the graduates and those Mho 
eliminate for flying deficiency reasons. 

These hypotheses were evaluated by comparing the means of the psychomotor scores among three 
categories: UPT graduates, all UPT ellmlnees, and UPT flying deficiency ellmlnees. Figure 3 
presents the differences among these groups. The actual means with their zero-order correlations 
to UPT outcome and tests of significance are listed In Appendix A. All five scores had 
significant differences (£ < .001) between graduates and either category of ellmlnees. In 
addition, for all three of the Complex Coordination scores, the tneans for flying deficiency 
ellmlnees were significantly worse (£ <.01) than those for individuals eliminated for all other 
reasons. These results suggest that both tests identify ellmlnees, but the Complex Coordination 
Test does a better job of identifying flying deficiency ellmlnees. These results validate the 
use of all five psychomotor scores as predictors of success or elimination in USAF UPT. 



Concurrent Validity 

The psychomotor testing of UPT students nearing graduation at Williams ARB permitted the 
concurrent validation of the tests against two criteria of UPT flying performance supplied by 
ATC. The first was the ATRB decision whether or not to recommend the student for follow-on 
assignment to a fighter-attack-reconnaissance (FAR) aircraft. Graduates who were not 
FAR- recommended would receive a follow-on assignment to a tanker-transport-bomber (TTB) 
air** aft. Only the better students received FAR recommendation. The second criterion was an 
experimental class ranking accomplished by the flight commanders of the UPT training squadron. 
Figure 4 presents all five means for both FAR- and non-FAR-recommended students. The FAR 
students had significantly (£ < .01) better scores than the non-FAR students on two of the five 
psychomotor measures (a third score was significant at ££.05). Also, using multiple linear 
regression, the five psychomotor scores had significant (jp 1 .01 ) multiple correlations against 
both the class ranking (R » .41) and the FAR/non-FAR (R ■ .43) criteria. These results show that 
in addition to identifying candidates with low probabilities of UPT success, psychomotor scores 
also relate to superior performance in UPT. Taken altogether, these results indicate that the 
quality of UPT students can be Improved with psychomotor screening. 



Psychomotor Screening Equation 

To obtain the maximum prediction accuracy from the psychomotor scores, a weighted equation or 
linear model was developed to predict UPT outcome. This equation provides a screening score 
based on psychomotor abi 1 1 ty. Because the cri teri on was di chotomous and was coded 0 for 
ellmlnees and 1 for graduates, the screening score can be roughly interpreted as a probability of 
success in UPT. To determine the weights for the model, multiple linear regression was used, 
with all five psychomotor scores as predictors and UPT outcome as the criterion variable. 

Although all five psychomotor scores were significant predictors of UPT outcome, the scores 
of each test were highly Interrelated, as shown by the factor analyses. Accordingly, the most 
useful linear model would contain the fewest psychomotor scores which still accounteo for as much 
of the criterion variance as all five scores together. However, that does not mean that the 
tests could be changed to present only the most predictive axes, because that would change the 
nature of the tasks in the tests. After several comparisons using the F-ratio, the final 
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Figure 3 . Nean Psychoaotor Scores by UPT Outcoae. 
(N > 1.725) 
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screening equation (R » .196) contained only the and Yg scores. (Appendix B sunmarlzes 
this analysis.) Therefore, the most useful screening score based on psychomotor ability can be 
computed from the following equation: 



• a + (b] X Xl^) + (bg x Y2^) 



where 



■ psychomotor screening score of 1th subject 
" constant (0.95648) 

« weight of psychomotor X] score (-0.0000075748) 

■ weight of psychomotor Yg score (-0.000020375) 
' psychomotor X] score of ith subject 

« psychomotor Yg score of 1th subject 



a 



bl 
b2 

Xli 
Y2i 



This psychomotor screening score was computed for the 1,725 cases in the developmental 
sample. The mean screening scores for the UPT graduates (79.0) and UPT eliminees (75.1) are 
depicted in Figure 5. Though the difference in the means was small, it was itatistically 
significant £.001). 



Psychomotor Screening System 

The next step in the analyses was to determine the practical value of these scores 
differences, "^o use this information in a selection system, the psychomotor screening scores 
could be computed from the psychomotor test results for all available candidates. The candidates 
could then be rank-ordered, based on the confuted screening scores; and the best candidates could 
be selected to fill the required training quotas, rigure 6 depicts such a ranking of the 1,725 
subjects of the development sample. The sample was segmented into decile groups (lOK each) for 
ease of comparison* The lowest-ranked decile is on the far left and the highest-ranked is on the 
far right. Each decile bar was further divided by the percentage of that decile that actually 
eTIminated from UPT. As shown by the trend across the deciles, the lower-ranked deciles had 
greater percentages of eliminees than did the deciles ranked higher by the psychomotor screening 
system. 



Utilization of Psychomotor Screening System 

If all available candidates could be compared simultaneously, then the seTection procedure 
could simply choose the number of candidates required from those with the highest screening 
scores. In operational use, this procedure would be complicated by the problems of multiple 
commissioning sources and the long-term planning required in thf> UPT selection process. An 
alternate approach would select only those candidates who obtain a screening score above a 
predetermined cutoff. The cutoff score would be determined and updated by comparing the 
recruiting environment and UPT attrition information. This approach would permit psychomotor 
screening to function as a "gate" at any step in the selection process in the same way that the 
AFOOT and FSP results are currently used (I.e., candidates with scores above the minimum cutoff 
proceed to the next phase of selection while those below the cutoff are rejected). 
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The higher the psychowtor screening cutoff score Is set, the greater the Impact on attrition 
In UPT. However, sone rejection of candidates who could pass training will also occur. This 
"false rtjectlon" Is a problem with all screening system, Including flying screening, and must 
be evaluated In tenis of the percentages screened and the quality of the graduating students. 
Stnce the psychoaotbr scores were related to Measures of UPT quality (class ranking and FAR 
reconendatlon). It Is likely that the potential graduates In the lower deciles who would be 
■false rtjectlons" would be poorer-perfomlng graduates who would require more Instruction and 
flying tim. 

Table 7 shows sd«e examples of cutoff scores with the corresponding results that would have 
occurred had the cutoff applied to the 1,725 cases of the development sample. For each cutoff 
score, the hit-miss classification provided the percentile of the entire sample which would be 
screened, the percentage of actual UPT attrltees In the sample who would be correctly screened 
and the percenUge of UPT graduates In the sample who would be falsely rejected. 

Table 7. Examples of Psychomotor Screening 



Cutoff scoro 
percentITt 


Z of attrltcts 
corractly njectad 


t of graduates 
Incorrectly rejected 


5th ' 


11.1 


3.5 


10th 


19.6 


8.0 


20th' 


S1.8 


15.7 


30th 


45.9 


26.7 



Cross-validation 

Results were cross-validated using 166 candidates who completed UPT In FY83. These were the 
latest data fro« UPT available at the tine of the analyses. The screening scores, conputed for 
the FY83 senile using the psychOMtor screening equation above, correlated significantly 
(£ 1 .005) with UPT outcoM (R • .23). The correlation was slightly larger than those found In 
the development saaples, but this saaple was considerably saaller. These results validate the 
psychomotor screening syste. on a nore recent SMple that was Independent of Its development. 
This suggests that psychomotor measures are sufficiently robust to be collected several years 
prior to UPT and still permit the psychomotor screening system to be effectively applied at UPT 
entry. 



IV. INTEGRATED SELECTION SYSTEM 

A potentially more effective approach to pilot candidate selection would consider all valid 
screening information simultaneously, Including psychomotor. The advantage of an Integrated 
apprdach Is that, 'through analytical techniques, the best mix of the Information from the 
multiple sources can be obtained, with coit>onents selectively weighted to Increase screening 
effectiveness. Such an Integrated system would contain a selection profile with the ranges of 
•11 the screening measures known for UPT graduaites. A candidate's scores on all of the screening 
measures would be simultaneously compared to the successful graduate profile to produce an 
overall predicted probab111t;y of tiPT success. 

Current operational USAF pilot selection considers information from the five composite scores 
of the AFOQT, age at the start of UPT, possession of a college technical degree, and possession 
of a private pilot's license (PPL) or completion of a USAF light aircraft flying program like the 
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FSP. Upon Investigation. USAF historical rtcords were found to be very Halted regarding the 
possession of a PPL and tje flying results of the AFA Pilot Indoctrination P.^ran (PIP) and the 
AFROTC Flight Instruction Prograa (FIP). Therefore, these data were not Included In the 
developnenUI analyses. The only flying InfonMtlon Included was the sUndardlzed results of the 
FSP conducted at Hondo AFB» Texas. The technical degrn InfonMtlon and FSP final outcoae 
InfonMtlon were used as d1chotOM)us variables and coded 1 or 0 as appropriate. Two continuous 
variables based on FSP results were also Included. These were the grad ^f the final evaluation 
flight and the sua of the grades fro« all of the other flights. In !on, gender and race of 
the candidates were Included In the analysis. Appendix C lists the i.ro*order correlations of 
these iMasures with the criterion (UPT graduati on/el 1a1 nation for all reasons), each derived for 
all available subjects fro« the historical (FY79 to FY83) records. 

Model Developaent 

Because pilot candidates are chosen differently, depending on their source of con1ss1on» the 
available sets of historical InfonMtlon were different for each coMlsslonlng source. 
Therefore, three Integrated Pilot Candidate Selection Models (IPCSMs) were developed. 
Specifically, the FSP results were available only for OTS candidates. In addition, because AFA 
candidates were not required to Uke the AFOQT, these scores were available only for OTS and 
AFROTC candidates. Each IPCSN was developed fro« the piychOMtor scores and the specific 
InfonMtlon currently used to select candidates from each Mjor coairisslonlng source. The 
saMples used to develop each IPCSN are presented In Table 8. As was done for the piychOMtor 
screening systea, the three IPCSMs were reduced te the saallest set of Measures possible without 
losing significant prediction, using F-ratlos to amf^r^ the results of the various Miltlple 
linear regression aralyses. 



IPCSM I (OTS Model). The 268 subjects for IPCSM I (OTS Model) had 17 Measures available 
(five psychonotor scores, age, five AFOQT conposlte scores, thrM FSP results, technical degree, 
gender, and race). This full aodel was reduced In a logical stepwise fashion down to a final 
seven-variable Mdel. The F-ratIo between the full aodel and the seven-variable final aodel 
(containing psychoMter and Y2 scores, age, AFOQT pilot and navlgater coi^oslte scores, 
and FSP sub of lesson grades and final grade) revealed no significant difference in prediction^ 
Appendix 0 contains a sumry of the analyses. In the presence of these Measures, gender, race, 
and possession of a technical degree did not add te the prediction of UPT outcoM. 

The final prediction Measures of IPCSM I were regressed against UPT outcoMO to produce a 
profile of a UPT graduate In the fom of a weighted equation. IPCSM I (OTS Model) will produce a 
score corresponding to the probability of UPT success using the following equation: 

- a ♦ (b^ X Xl^) ♦ (b2 x Y2^) 4 (bj x age^ ) 

♦ (b4 X AFOQTP^) ♦ (b5 X AFOQTN^) 

♦ (bg X FSPl^) ♦ (by X FSP2^) 



Table 8. SaMple Sizes for Integrated 
Pilot Candidate Selection Models 



IPCSMs 



SOMple Size 



IPCSM I (OTS Model) 
IPCSM II (AFROTC Model) 
IPCSM III (AFA Model) 



268 
753 
310 
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where Yf 


- IPCSN I Score for 1th subject 


a 


■ constant (0.10192) 




■ weight for psychomotor Xl score (-.000015211) 




- weight for psychooiotor Y2 score (-.000010618) 


h 


■ weight for age of subject (-.0012) 


H 


■ weight for AFOQTP/AFOQT pilot score (0.00112) 




■ weight for AFOQTN/AFOQT navigator score (0.00029067) 




■ weight for FSPl/FSP final grade (0.08249) 


h 


■ weight for FSP2/FSP sun of grades (0.01923) 


XI i 


■ psychomotor X\ score of 1th subject 




• psychomotor Y2 score of 1th subject 


age^ 


■ age at start of UPT of 1th subject 


AFOQTP^ 


■ AFOQT pilot composite score of 1th subject 


AFOQTNf 


■ AFOQT navigator composite score of 1th subject 


FSPIf 


■ FSP final grade of 1th subject 


FSP2f 


■ FSP sum of grades of 1th subject 



IPCSM II (AFROTC Model) . Similar analyses were performed for IPCSM II (AFROTC Model) using 
753 subjects who had 14 measures available (five psychomotor scores, age, five AFOQT composite 
scores, technical degree, gender, and race). The F-ratIo between the full model and a final 
five-variable model (psychomotor X] and Yp scores, age, and AFOQT pilot and navigator 
composite scores) revealed no significant difference In prediction. Appendix 0 provides a 
summary of the analyses. Again, technical degree, gender, and race were found to be redundant In 
the presence of ps^ychomotor, age» and AFOQT Information. 

The final prediction measures of IPCSM II were regressed against UPT outcome to produce a 
weighted profile equation. IPCSM II will produce a score corresponding to the probability of UPT 
success score using the following equation: 

Yf - a ^ (b] X Xlf) ♦ (b2 x Y2f) ♦ (b3 x age^ ) 
^ (b4 X AFOQTPf ) * ib^ x AFOQTN^) 



where 



a 

h 



IPCSM II Score for 1th subject 
constant (1.72485) 

weight for psychomotor XI score (-.0000030409) 
weight for psychomotor Y2 score (-.000022526) 
weight for age of subject (-.04517) 
weight for AFOQTP/AFOQT pilot score (0.00124) 
weight for AFOQTN/AFOQT navigator score (0.00225) 



The common variables are as defined In the IPCSM I equation. 

IPCSM III (AFA Model). Finally, the 310 subjects for IPCSM III (AFA Model) had eight 
measures available (five psychomotor, age, technical degree, and race). Gender was not Included 
because all AFA candidates In this sample were males. The F-ratIo between the full model and a 
final four-variable restricted model (psychomotor X] and Y2 scores, age, and technical 
degree) revealed no significant difference In prediction. Appendix D shows a sunnary of these 
analyses. Without AFOQT data available, technical degree was found to be a significant predictor 
In IPCSM III (AFA Model). 

The final prediction measures of IPCSM III were regressed against UPT outcome to produce a 
weighted profile equation. IPCSM III will produce a score corresponding to the probability of 
UPT success score using the following equation: 
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- a + (bi X Xl^) + (b2 x Y2i ) ♦ (63 x age^ ) + (b^ x tech^) 



where 



IPCSM III Score of 1th subject 
constant (1.43056) 

weight for psychomotor XI score (*• 000005081 8) 
weight for psychomotor Y2 score (-.0000072309) 
welgf. for age of subject (-.02111) 
weight for possession of technical degree (.08756) 
1 If 1th subject has a technical degree, 0 otherwise 



a 



tech^ 



Common variables are as defined In the IPCSM I equation. 



Predictive Yilldatlon 

The weighted equations of the three IPCSMs were used to generate Integrated selection scores 
for the pilot candidates of the three development samples within each IPCSM classification. 
These selection scores correspond to a candidate's probability of UPT success as predicted by the 
measures In IPCSM, The differences of the mean predicted scores between UPT graduates and 
ellmlnees for all three IPCSMs, and the correlations between these scores and UPT outcome, were 
significant (££.01) and are shown by Figure 7, These results validate the use of all three 
IPCSMs as predictors of success or elimination In USAF UPT, 



Utniiitlon of the IPCSM Systems 

As was done for the psychcmotor screening system, the practical value of the significant 
differences In scores between graduates and ellmlnees was demonstrated for the three samples by 
separately ranking them on their IPCSM scores. For example, the 268 subjects In the IPCSM I (OTS 
Model) sample were rank-ordered and divided Into deciles as depicted In Figure 8, Again, the 
lowest-ranked decile Is on the far left, the highest-ranked Is on the far right, and the bottom 
part of each decile bar Is further divided by the percentage of that decile that actually 
eliminated from UPT. 

The trend toward Increasing percentages of ellmlnees within each higher-ranked decile Is 
steeper In IPCSM I than for the psychomotor-only screening model (Figure 6) because of the 
Increased accuracy obtained by Integrating the selection Information. Though not presented, the 
corresponding plots for IPCSMs II and III were similar. 

Any of the three IPCSMs could be used to select UPT candidates In the same manner suggested 
for the psychomotor screening system. A predetermined cutoff score would form the screening 
point. Tables 9, 10, and 11 list examples of cutoff scores for the three IPCSMs with the results 
that would have occurred had the cutoff applied to the corresponding cases of the development 
samples. For each cutoff score, the hit-miss classification provided the percentile of the total 
sample which would have been screened, and the percentage of actual UPT attrltees and percentage 
of UPT graduates at or below the cutoff. Each IPCSM would have correctly screened the attrltees 
and Incorrectly rejected the potential graduates by the Indicated percentages. 

Both the percentages of ellmlnees correctly screened and potential graduates who would be 
Incorrectly screened showed improvement over the psychomotor screening system at every selection 
score cutoff point in these examples. Each IPCSM system, because of the Integration of all of 
the available Information, appears to Identify more ellmlnees and screen fewer potential 
graduates than use of the same Information In a series of gates. 
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Figure 7. Mean IPCSN Scores by Ul>T OutcoM. 



ERIC 



27 



0) 

□ 



LU 

o 

LU 
0. 



100 



80 - 



60 - 



40 - 



20 - 




1ST 2ND 3RD 4TH 5TH 6TH 7TH 8TH 9TH 10TH 
LOWEST SCORES HIGHEST SCORES 



DECILE GROUPS 
RANKED BY INTEGRATED SYSTEM 



ERIC 




ELIMS 



□ 



PRADS 



Figure 8. Decile Groups Ranked by IPCSM I (OTS Model). 
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Table 9. Examples of IPCSM I (OTS Model) Screening 



Cutoff score 


X of attrltees 


X of graduates 


percentile 


correctly rejected 


Incorrectly rejected 


5th 


18.4 


0.0 


10th 


27.6 


2.6 


20th 


47.4 


10.4 


30th 


57.9 


19.3 


Table 10. 


Examples of IPCSM II (AFROTC Model) Screening 


Ciitoff score 


X of attrltees 


X of graduates 


percentile 


correctly rejected 


Incorrectly rejected 


5th 


15.0 


2.5 


10th 


27.2 


6.6 


20th 


38.8 


15.7 


30th 


54.4 


26.2 


Table 11. 


Examples of IPCSM III (AFA Model) Screening 


Cutoff score 


X of attrltees 


X of graduates 


percentile 


correctly rejected 


Incorrectly rejected 


5th 


16.1 


4.3 


10th 


32.3 


8.6 


20th 


48.4 


15.8 


30th 


51.6 


26.2 



Cross*Vi11dit1on 

The FY83 UPT sample was again used to cross-validate the three IPCSMs. IPCSM I Integrated 
scores were coiq)uted for the 82 subjects with all seven measures available and then correlated (R 
- .33) with UPT outcome. Though some shrinkage In correlation occurred for IPCSM I, the 
relationships were still significant (£> .001). For IPCSM II, 64 subjects had five measures 
available which significantly correlated (R * .33, £ > .005) with UPT outcome. Thus, IPCSMs I 
and II cross-validated as predictors of UPT outcome on a more recent sample which was Independent 
of their development. Finally, 165 subjects had the four measures of IPCSM III which correlated 
(R " .11) with UPT outcome. This correlation was not significant (£ < .07), though the 
correlation In the development sample was; and questions about the general Izablllty of IPCSM III 
remain unanswered. 



V. CONCLUSIONS/RECOMMENDATIONS 

ine results of the analyses In this report demonstrate that the two psychomotor tests 
generated by the computer-based test devices, Two-Hand Coordination and Complex Coordination, 
produce scores that are valid predictors of UPT outcome. Candidates who are likely to graduate 
from UPT can be differentiated by their scores from those who are likely to eliminate. Also, 
superior UPT students (FAR-recommended) can be differentiated from weaker students (non-FAR). 

The differentiation of pilot candidates could be obtained through the Implementation of the 
p$ychomotor-on1y screening system In the fashion of other screening gates currently being used 
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(e.g.. AFOQT. FSP). Since the cross-validation results demonstrated the robustness of the 
psychomotor Measures over time, psychomotor testing could be administered very early in the 
selection process and still significantly predict the Individual's probability of success without 
the need to retest the candidate at UPT entry. The strength of prediction and costs of 
psychomotor testing would be expected to be greater than those associated with the administration 
of the AFOQT but less than for flying screening. This could determine the logical sequence for 
these three screening gates. To maximize the impact on attrition and the quality of the students 
selected for UPT, the highest possible psychomotor screening scores should be selected »s the 
cutoffs. This policy would screen the largest percentage of the candidates who would attrite. 
However, many potential graduates would also be rejected. 

Psychomotor scores are not the only important Information which should be used in the 
selection of pilots; rather, all significant predictors should be utilized. The IPCSMs 
capitalize on the utility of all available information, including psychomotor measures, to 
predict success or failure in UPT as well as quality of performance as measured by FAR 
recommendations. The "Whole Person Concept" (an appraisal of the important strengths of a 
candidate) is imbedded in this multi-dimensional IPCSM approach to pilot selection. The IPCSMs 
will not only impact attrition and quality in UPT, but will also reduce the rejection of 
potential graduates. Also, IPCSMs I and II can be used to screen minorities and women without 
consideration of race or gender. 

Although the samples used in the development of the IPCSMs were aligned with a particular 
source of conission, that restriction was data dependent. The cross-validation of IPCSMs I and 
II were not restricted by source of commission. Therefore, the choice of the appropriate 
selection model to use should depend on the information available. IPCSM I is the strongest 
system (highest correlation with UPT outcome) and is recommended for use as the final selection 
system for candidates going through FSP (traditionally OTS cadets and recently, AFROTC cadets 
attending Summer Field Training at Lackland AFB, Texas). IPCSM II could be used as a 
preselection input either for OTS or for AFROTC Field Training. 

Unfortunately, IPCSM III does not appear sufficiently stronger in prediction than the 
psychomotor screening system alone and therefore does not warrant implementation in its current 
form. However, if objective measures similar to the FSP and AFOQT information were available for 
AFA candidates in addition to the currently available data and psychomotor scores, then results 
similar to those from IPCSM I could be expected. A follow-on R4D effort to obtain these measures 
for Acadeiiy cadets and validate an IPCSM system for them is recommended. 

Though the systems presented in this report will improve the selection of candidates for UPT. 
further refinements are possible. One improvement would develop the weights of the selection 
equations, not using the dichotomous UPT pass/fail criterion but instead,, using a continuous 
measure of the individual's flying performance both in training and on operational missions. The 
candidates thus selected for training would have good chances of becoming successful operational 
pilots. The data to produce a Pilot Performance Index to provide this enhancement are being 
collected by AFHRL, and these analyses are planned for the near future. Another improvement to 
these systems would include information on candidates who possess a PPL. Research which compares 
the psychomotor test performance of FSP graduates and candidates with PPLs is recommended to 
extend the general izabi 11 ty of these selection systems. 

The addition of psychomotor screening to the selection procedure does not preclude the search 
for other valid measures of flying aptitude. Every valid measure added to an integrated 
selection procedure would help reduce the attrition and false rejections in training. A 
promising research area is the measurement of mental aptitudes such as Information processing and 
decisiveness. The advent of extremely powerful microcomputers makes possible the measurement of 
reaction time and accuracy on very complex mental tests. Including such validated measures in 
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the IPCSM systMS would ii^rove tlie current results. AFHRL has developed a battery of wntal and 
personality tests, called the Basic Attributes Tests (BAT), for this expressed purpose. These 13 
tests (iBhoff a Levine, 1981) are adninistered by a highly portable and flexible test device, 
called the Porta-BAT. Results of this ongoing effort will be reported in the future. 

In conclusion, the in)le«entation of psychOBWtor screening, either as a separate gate or 
within an integrated selection systea. will i«prove the quality of student candidates in USAF 
UPT. Based on the results of this current effort, an integrated selection systea including 
psychomotor screening is recoiiKnded. Future research should extend both the selection 
Infomiation being considered and the criteria to be predicted. 
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LIST OF ABBREVIATIONS 



AFA 


Air Force Acadeiny 


AFB 


Air Force Base 


AFHRL 


Air Force Human Resources Laboratory (AFSC) 


AFOQT 


Air Force Officer Qualifying Test 


AFROTC 


Air Force Reserve Officer Training Corps 


AFSC 


Air Force Systems Conrnand 


ATC 


Air Training Coitmand 


ATRB 


Advanced Training Reconmendatlon Board (ATC) 


BAT 


Basic Attributes Tests 


CRT 


Cathode*Rj^ Tube 


FAR 


Fighter-attack-reconnaissance (training track) 


FIP 


Flight Instruction* Program (AFROTC) 


FSP 


Flight Screening Program (OTS) 


IPCSM 


Integrated Pilot Candidate Selection Model 


OTS 


Officer Training School 


PIP 


Pilot Indoctrination Program (AFA) 


PPL 


Private Pilot's License 


RPR 


Request for Personnel Research 


R«D 


Research and Development 


TTB 


Tanker- transport-bomber (training track) 


UPT 


Undergraduate Pilot Training 


USAF 


United States Air Force 
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APPENDIX A: COMPARISON OF PSYCHOMOTOR SCORES BY UPT OUTCOME 



Psychoaotor Scores 









l2 


12 


l2 


Correlation 
witn UPT outcomeiN * ifjco) 


-.12 


1 n 
-.10 


1 A 

-.14 


-.17 


-.15 


Means by UPT Outcome^ 












Graduate Means (N - 1,348) 


14,315 


16,341 


3,559 


2,858 


4,725 


All El In Heans(N " 377) 


15,829 


17,621 


4,936 


4,173 


6,678 


Flying Deficiency (FD) 
El In Means (N - 199) 


16,302 


18,007 


5,593 


4,702 


7,580 


Probabilities of Psychoax>tor Score Differences Occurring by Chance Alone 




Grads vs. All El Ins 


.001 


.001 


.001 


.001 


.001 


Grads vs. FD El Ins 


.001 


.001 


.001 


.001 


.001 


FD vs. Other El Ins 


.103 


.182 


.006 


.010 


.005 



•Psychonotor scores reflect errors; therefore, lower scores mean better test 



perfomance. 




APPENDIX B; PSYCHOMOTOR SCREENING MODEL REDUCTION 



F-RATIO TEST (N ■ 1.725) 

Psychoaotor Indtpcndent 

, v«r1*blM prtdlctors R^ p value' 

Full Mo<le1 X1,Y1,X2,Y2,Z2 6 . 04225 . 0000 

Restricted Model XI, Y2 3 . 03842 . 0000 



(Rf - r|) / (IP| - IP2) 
^ ' ( 1 - R2 1 ) / ( N - IP, ) 



P _ (.04225 - .03827) / (6 - 3) 
( 1 - .04225 ) / ( 1725 - 6 ) 

F ■ 2.38 

Mote. No significant difference In prediction strength of these two 

■odtis since coaputed F-rat1o value docs not exceed the critical F value 

(Fc - 3.78). 

'Probability of R^ Occurring by Chance Alone. 
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APPENDIX C ; ZERO-ORDER CORRELATIONS OF IPCSM PREDICTORS WITH UPT OUTCOME 



Predictors 


Correlation trtth UPT 






Dtss/fall 


N« 


FSP Pass/Fall 


.137 


1534 


FSP Final Grade 


.271 


1534 


FSP Sun of Grades 2-11 


.303 


1534 


AFOQT Pilot score 


.158 


4460 


AFOQT Navigator Technical 


.148 




score 






AFOQT Acadenic score 


.080 


4577 


AFOQT Verbal score 


.007 


4576 


AFOQT Quantitative score 


.138 




Psychonotor XI score 


-.118 


1918 


PsychQMtor Y1 score 




101 0 


Psychonotor X2 score 


-.153 


1906 


Psychonotor Y2 score 


-.181 


1910 


Psychonotor Z2 score 


-.146 


1916 


Age 


-.120 


8438 


Gender 


.033 


8438 


Race 


.110 


8292 


Possession of Tech Degree 


.111 


8183 



*The sanple sizes (N) were detemlned for all cases with valid 
data available for the prediction variable and the UPT criterion f ron 
the FY79 to FY83 UPT historical records. 
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APPENOIX D; IPCSMs REDUCTION USING F-RATIO TESTS 



Full Model 



Restricted Model 



(R^ 1 - 2) / ■ ^P^) 
(1 - R2 ,) / (N - IPl) 



IPCSM I Variables 



Psychomotor (5) 
FSP(3) 
AF0QT(5) 
Age 

Tech Degree 

Race 

Gender 



Psychoinotor(2) 
FS?(2) 
AFDQT(2) 
Age 



Analyses results 



r2 - .23254 
df ■ 17 
N - 265 
£ < .DDOD 



r2 « .215D1 
df « 7 
N ^ 265 
£ < .0000 



IPCSM II variables 



Psychoinotor(5) 

AFD0T(5) 

Age 

Tech Degree 

Race 

Gender 



Psychomotor(2) 

AFD01(2) 

Age 



F-ratIo test calculations 



- ^ (,23254 ■ ,215D1) / (18 - 8) 
" (1 - .23254) / (265 - 18) 



F ■ D.56* 
F(crltlcal) ■ 2.41 



F-ratIo test calculations 

P . (.09673 - .08437) / (15 - 6) 
(1 - .09673) / (741 - 15) 

F » 1.10* 

F(crltlcal) ■ 2.41 



Analyses results 



r2 ■ .09673 
df ■ 15 
N ■ 741 

£ . DODO 



r2 - .08437 
df ■ 6 
N « 741 
2 < .0000 



IPCSM III variables 



Psychomotor (5) 
Tech Degree 
Age 
Race 



Psychomotor (2) 
Tech Degree 
Age 



F » 



F-ratIo test calculations 



(.05581 - .04294) / (9 - 5) 
(1 - .05581) / (305 - 9) 



1.01^ 



Analyses results 



r2 ■ .09673 
df ■ 15 
N ■ 741 

.0000 



r2 ■ .08437 
df ■ 6 
N ■ 741 

2 < .0000 



F(crltlcal) ■ 3.32 



*No significant difference In models. * U-S. government printing office: 1986- 6 59-055 / ko02 0 
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